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SECTION  1.  GENERAL  INFORMATION 


1.1  BACKGROUND 

Technologies  under  development  for  the  detection  and  discrimination  of  munitions  and 
explosives  of  concern  (MEC),  i.e.,  unexploded  ordnance  (UXO)  and  discarded  military 
munitions  (DMM),  require  testing  so  their  perfonnance  can  be  characterized.  To  that  end,  the 
U.S.  Army  Aberdeen  Test  Center  (ATC)  located  at  Aberdeen  Proving  Ground  (APG),  Maryland, 
has  developed  a  Standardized  Shallow  Water  Test  Site.  This  site  provides  a  controlled 
environment  containing  varying  water  depths,  multiple  types  of  ordnance  and  clutter  items,  as 
well  as  navigational  and  detection  challenges.  Testing  at  this  site  is  independently  administered 
and  analyzed  by  the  government  for  the  purposes  of  characterizing  technologies,  tracking 
performance  during  system  development,  and  comparing  the  performance  and  costs  of  different 
systems. 

The  Standardized  UXO  Technology  Demonstration  Site  Program  is  a  multiagency 
program  spearheaded  by  the  U.S.  Army  Environmental  Command  (USAEC).  ATC  and  the 
U.S.  Army  Corps  of  Engineers  Engineering,  Research  and  Development  Center  (ERDC)  provide 
programmatic  support.  The  Environmental  Security  Technology  Certification  Program 
(ESTCP),  the  Strategic  Environmental  Research  and  Development  Program  (SERDP),  and  the 
Army  Environmental  Quality  Technology  Program  (EQT)  provided  funding  and  support  for  this 
program. 

1.2  OBJECTIVE 

The  objective  of  the  Shallow  Water  Standardized  UXO  Technology  Demonstration  Site  is 
to  evaluate  the  detection  and  discrimination  capabilities  of  existing  and  emerging  technologies 
and  systems  in  a  shallow  water  environment.  Specifically: 

a.  To  determine  the  demonstrator’s  ability  to  survey  a  shallow  water  area,  analyze  the 
survey  data,  and  provide  a  prioritized  “Target  List”  with  associated  confidence  levels  in  a  timely 
manner. 

b.  To  determine  both  the  detection  and  discrimination  effectiveness  under  realistic 
scenarios  that  varies  ordnance,  clutter,  and  bathymetric  conditions. 

c.  To  determine  cost,  time,  and  manpower  requirements  needed  to  operate  the  technology. 

1.3  CRITERIA 

The  scoring  criteria  specified  in  the  Environmental  Quality  Technology  -  Operational 
Requirements  Document  (EQT-ORD)  (app  D,  ref  1)  for:  A(1.6.a):  UXO  Screening,  Detection 
and  Discrimination  document  are  presented  in  Table  1-1.  Very  little  infonnation  was  available 
on  the  capabilities  of  shallow  water  detection  systems  when  these  criteria  were  developed. 
However,  they  were  used  in  the  design  of  the  test  site,  and  the  five  metrics  were  used  to  measure 
system  perfonnance  in  this  report. 
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TABLE  1  - 1 .  SCORING  CRITERIA 


Metric 

Threshold 

Objective 

Detection 

80%  ordnance  items  buried  to 

1  foot  and  under  8  feet  (2.4  m)  of 
water  at  a  standardized  site 
detected 

95%  ordnance  items  buried  to 

4  feet  and  under  8  feet  (2.4  m)  of 
water  at  a  standardized  site 
detected 

Discrimination 

Rejection  rate  of  50%  of 
emplaced  non-UXO  clutter  at  a 
standardized  site  with  a  maximum 
false  negative  rate  of  1 0% 

Rejection  rate  of  90%  of  emplaced 
non-UXO  clutter  at  a  standardized 
site  with  a  maximum  false 
negative  rate  of  0.5% 

Reacquisition 

Reacquire  within  1  meter 

Reacquire  within  0.5  meter 

Cost  rate 

$4000  per  acre 

$2000  per  acre 

Production  rate 

5  acres  per  day 

50  acres  per  day 

The  ATC  shallow  water  site  is  designed  to  evaluate  the  threshold  detection  level  of  a  range 
of  ordnance  at  the  1-foot  +  8-foot  requirement.  Limited  information  is  available  at  the  objective 
detection  level.  All  other  measured  results  in  this  test  were  evaluated  against  both  criteria  levels. 

1 .4  APG  SHALLOW  WATER  SITE  INFORMATION 

1.4.1  Location 


The  Aberdeen  Area  of  APG  is  located  in  the  northeast  portion  of  Maryland  on  the  western 
shore  of  the  Chesapeake  Bay  in  Harford  County.  The  Shallow  Water  Test  Site  is  located  within 
a  controlled  range  area  of  APG. 

1.4.2  Soil  Type 

The  area  chosen  for  the  shallow  water  test  site  was  known  as  Cell  No.  3  in  a  dredge-spoil 
field.  The  cell  bottom  is  composed  primarily  of  sediment  removed  from  the  Bush  River.  This  is 
a  freshwater  site. 

1.4.3  Test  Areas 


a.  The  test  site  contains  five  areas:  calibration  grid,  blind  test  grid,  littoral,  open  water, 
and  deeper  water.  Additional  detail  on  each  area  is  presented  in  Table  1-2.  A  schematic  of  the 
calibration  lanes  is  shown  in  Figure  1. 
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TABLE  1-2.  TEST  AREAS 


Area 

Description 

Calibration  grid 

The  calibration  area  contains  15  projectiles,  3  each  40,  60,  81,  105,  and  155  mm. 

One  of  each  projectile  type  is  buried  at  the  projectile  diameter  to  depth  ratio  shown 
in  Figure  1.  This  area  is  designed  to  provide  the  user  with  a  sensor  library  of 
detection  responses  for  the  emplaced  targets  and  an  understanding  of  their  resistivity 
prior  to  entering  the  blind  test  fields.  Two  “clutter- cloud”  target  scenarios  have  been 
constructed  adjacent  to  this  area  (fig.  1). 

Blind  grid 

The  blind  grid  contains  644  detection  opportunities.  Each  grid  cell  is  2  x  2  m2.  At 
the  center  of  each  cell  is  either  an  ordnance  item,  clutter,  or  nothing.  Surrounding 
the  blind  grid  on  three  sides  are  3.6-kg  (8-lb)  shot  puts,  buried  0.3  meters  deep  in  the 
sediment.  The  shot  puts  can  be  used  as  a  navigational/Global  Positioning  System 
(GPS)  check.  The  GPS  coordinates  for  the  center  of  each  grid  and  the  shot  put 
locations  are  provided  to  the  vendor  prior  to  testing. 

Littoral 

This  is  a  sloping  area  on  one  side  of  the  pond  with  vegetation  growing  into  the 
waterline.  Water  depth  ranges  from  0.3  to  1.8  meters.  It  contains  a  variety  of 
navigational  and  detection  challenges. 

Open  water 

The  open  water  scenario  contains  a  variety  of  navigational,  detection,  and 
discrimination  challenges.  Water  depth  varies  from  1.8  to  3.4  meters. 

Deeper  water 

The  water  depth  in  this  area  varies  between  3.4  and  4.3  meters. 

4X4  meters 


155-mm 

155-mm 

155-mm 

• 

• 

• 

1:7 

1:5 

1:1 

3X3  meters 


105-mm 

105-mm 

105-mm 

• 

• 

• 

1:1 

1:5 

1:8 

40-mm 

40-mm 

40-mm 

60-mm 

60-mm 

60-mm 

81-mm 

81-mm 

81-mm 

• 

• 

• 

• 

• 

• 

• 

• 

• 

1:11 

1:5 

1:1 

1:1 

1:5 

1:11 

1:1 

1:5 

1:7 

2X2  meters 


Centered  clutter-cloud  Offset  clutter-cloud 


Figure  1.  Schematic  of  the  calibration  grid. 
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b.  The  water  depth  at  this  facility  during  testing  is  maintained  such  that  the  calibration  and 
blind  grid  areas  meet  the  2.4-meter  (8-ft)  detection  criterion  specified  in  section  1.3.  The  test  site 
is  approximately  2.8  hectares  (6.9  acres)  in  size. 

1.5  GROUND  TRUTH  TARGETS 

The  ground  truth  is  composed  of  both  inert  ordnance  and  clutter  items.  The  inert  ordnance 
items  are  listed  in  Table  1-3.  All  items  were  located  in  storage  sites  at  APG.  The  items  have  not 
been  fired  or  degaussed. 

Clutter  items  fit  into  one  of  three  categories:  ferrous,  nonferrous,  and  mixed  metals.  The 
ferrous  and  nonferrous  items  have  been  further  divided  into  three  weight  zones  as  presented  in 
Table  1-4,  and  distributed  throughout  all  test  areas.  Most  of  this  clutter  is  composed  of  ordnance 
components;  however,  industrial  scrap  metal  and  cultural  items  are  present  as  well.  The 
mixed-metals  clutter  is  composed  of  scrap  ordnance  items  or  fragments  that  have  both  a  ferrous 
and  nonferrous  component  and  could  reasonably  be  encountered  in  a  range  area.  The 
mixed-metals  clutter  was  placed  in  the  open  water  area  only. 


TABLE  1-3.  INERT  ORDNANCE  TARGETS 


Description 

Length, 

mm 

Diameter, 

mm 

Aspect 
Ratio,  W/L 

Weight,  g 

40-mm  L70  projectile 

208 

40 

0.1923 

965 

60-mm  mortar  M49A2 

185 

60 

0.3243 

975 

8 1  -mm  mortar  M3  74 

528 

81 

0.1534 

3969 

8 1  -mm  mortar  M82 1 

510 

81 

0.1588 

3338 

105-mm  projectile  Ml 

445 

105 

0.2360 

13834 

155-mm  M107  projectile 

684 

155 

0.2266 

41731 

8-in.  M104/106 

856 

203 

0.2371 

89811 

TABLE  1-4.  CLUTTER  WEIGHT  RANGES 


Clutter  Type 

Weight  Range  in  Grams 

Small 

Medium 

Large 

Ferrous 

10  to  510 

511  to  2200 

>2201 

Nonferrous 

10  to  270 

275  to  800 

>801 
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SECTION  2 .  SYSTEM  UNDER  TEST 


2. 1  DEMONSTRATOR  INFORMATION 

NAEVA  provided  the  information  in  sections  2.1  through  2.6  as  part  of  their  Broad 
Agency  Announcement  (BAA)  proposal  (app  D,  ref  2).  Section  2.8  contains  ATC’s  comments 
on  the  demonstrated  system. 

Note:  The  provided  demonstrator  infonnation  has  been  edited  to  comply  with  government  report 
guidelines. 

2.2  SYSTEM  DESCRIPTION 

a.  For  this  demonstration,  NAEVA  proposed  to  work  with  XTECH  to  deploy  the 
multisensor  underwater  system  using  Geonics  EM61  MKII  (underwater  coils)  electromagnetic 
(EM)  metal  detectors.  The  system  was  relatively  lightweight,  requiring  a  small  aluminum  boat 
for  towing.  This  configuration  should  have  allowed  the  team  to  achieve  full  coverage  of  the  site, 
even  in  relatively  shallow  areas.  Accurate  data  positioning  was  achieved  using  a  real-time 
kinematic  GPS. 

b.  The  deployed  system  (fig.  2)  consisted  of  two  underwater  coils  mounted  side  by  side  on 
a  specially  designed  acrylonitrile-butadiene-styrene  (ABS)  sled.  A  GPS  mast,  centered  over  the 
two  coils,  was  attached  directly  to  the  sled  and  allowed  accurate  positional  tracking  of  the  sensor 
data.  The  unit  was  towed  by  a  14-foot  aluminum  boat  powered  by  an  outboard  motor  with  a 
specialized  prop.  The  custom  fabrication  and  rigging  designed  for  the  system  allowed  excellent 
boat  control  and  maneuverability  while  towing.  During  data  collection,  the  coil  assembly  glided 
across  the  bottom  on  the  smooth  plastic  underbelly  of  the  sled. 
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Figure  2.  XTECH  sled  with  EM61  coils. 


2.3  DEMONSTRATOR’S  POC  AND  ADDRESS 

POC:  Mr.  Alexander  Z.  Kostera 

email:  akostera@naevageophysics.com 

Address:  NAEVA  Geophysics,  Inc. 

P.O.  Box  7325 
Charlottesville,  VA  22906 
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2.4  DEMONSTRATOR’S  SITE  SURVEY  METHOD 


NAEVA  intended  to  fully  map  the  Shallow  Water  Standardized  Test  Site,  using  XTECH’s 
dual-sensor  underwater  detection  system.  Mapping  activities  included  the  calibration  lanes, 
blind  grid  area,  and  open  water  sites,  including  both  the  deep  water  and  littoral  zones  (fig.  3). 
All  field  areas  were  surveyed  in  the  prescribed  order  in  a  single  orientation 
(e.g.,  north-south,  east-west).  If  time  permitted,  NAEVA  could  elect  to  remap  certain  portions 
or  the  entire  site  in  a  second  orientation  to  enhance  the  data  quality. 


Figure  3.  XTECH  sled  in  the  open  water  area. 


2.5  DEMONSTRATOR’S  QC  AND  QA 

a.  For  purposes  of  this  proposal,  QA  is  defined  as  the  procedures  to  be  employed  during 
the  demonstration. 

b.  All  geophysical  data  were  collected  with  real-time  GPS  data  positioning  from  an 
antenna  mounted  above  the  two  coils.  EM  data  were  collected  at  a  rate  of  10  readings  per 
second,  which  equates  to  more  than  one  reading  per  foot.  GPS  locations  were  logged  at  a  rate  of 
one  reading  per  second.  To  maintain  straight-line  profiling  and  to  minimize  the  occurrence  of 
gaps  within  the  data,  real-time  sensor-tracking  software  was  used.  The  Trimble  Agl70 
navigation  system  includes  a  light-up  display  mounted  in  the  boat  that  indicates  the  direction  and 
degree  of  correction  necessary  to  maintain  a  straight  path.  Positional  data  supplied  for  the 
calibration  lanes  and  blind  grid  area  are  overlaid  on  the  track  map  to  ensure  that  full  site 
coverage  has  been  achieved.  Although  the  GPS  has  a  listed  accuracy  of  3  cm,  the  expected 
accuracy  of  resultant  target  selections  was  signified  by  a  circle  with  a  1-foot  radius  around  each 
target. 
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c.  To  establish  confidence  in  the  data  reliability,  tests  were  conducted  in  a  systematic 
manner  throughout  the  duration  of  the  fieldwork.  Various  types  of  QC  data  were  generated 
before,  during,  and  after  ah  data  collection  sessions. 

d.  Daily:  A  location  was  identified  that  had  no  subsurface  metal  and  was  designated  as  a 
calibration  point.  Readings  were  collected  in  a  stationary  position  over  the  calibration  point  to 
ensure  that  a  stable  and  repeatable  response  was  exhibited.  This  test  was  performed  twice  daily 
to  establish  that  the  instrument  was  functioning  properly,  as  indicated  by  a  stable  and  repeatable 
response. 


e.  A  line  containing  at  least  one  seeded  item  was  identified  within  the  calibration  lanes 
that  served  as  a  standard  response  and  latency  check.  At  the  start  and  end  of  each  field  day,  two 
lines  were  collected  bidirectionally  across  the  item  using,  as  close  as  possible,  the  same  line  path. 
The  data  were  then  reviewed  for  consistent  response  and  positioning  and  to  detennine  an 
appropriate  latency  correction. 

f.  During  data  collection:  On  completion  of  the  original  collection  of  a  data  set, 
approximately  5%  of  the  line  footage  for  each  surveyed  area  was  re-collected  as  a  check  of 
instrument  repeatability  and  positioning.  The  repeat  lines  were  saved  to  separate  files  and  used 
to  create  profiles  that  provided  a  direct  comparison  with  the  original  data.  Each  profile  was 
evaluated  for  repeatability  in  both  instrument  response  and  data  positioning. 

2.6  DATA  PROCESSING  DESCRIPTION 

a.  The  geophysical  data  were  temporarily  stored  in  the  system’s  integrated  logger  during 
data  collection  and  then  downloaded  into  a  laptop  computer  for  on-site  review  and  editing. 
Using  Geosoft’s  Oasis  Montaj  software,  a  track  plot  of  the  instrument’s  GPS  positions  was 
created  to  ensure  that  adequate  data  coverage  had  been  achieved.  Preliminary  contour  maps 
were  created  for  field  review  of  each  survey  area.  Once  in-field  processing  and  review  were 
completed,  the  data  were  electronically  transferred  to  NAEVA’s  Virginia  office  for 
analysis/target  selection. 

b.  Geosoft’s  Oasis  Montaj  UXO  software  package  was  used  to  post  process  and  contour 
the  raw  data  and  to  identify  potential  UXO  targets.  The  program  identifies  peak  amplitude 
responses  of  the  frequency  associated  with,  but  not  limited  to,  UXO  items.  Anomalies  may  have 
generated  multiple  target  designations  depending  on  individual  signature  characteristics. 

c.  Geophysical  data  processing  included  the  following: 

(1)  Instrument  drift  correction  (leveling). 

(2)  Lag  correction. 

(3)  Digital  filtering  and  enhancement  (if  necessary). 

(4)  Gridding  of  data. 
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(5)  Selection  of  all  anomalies. 


(6)  Selection  of  targets  for  intrusive  characterization. 

(7)  Preparation  of  geophysical  and  target  maps. 

c.  Final  target  lists  for  the  three  scenarios  will  be  prepared  separately  in  the  specified 
formats  and  then  submitted  for  scoring. 

2.7  DEMONSTRATOR’S  SITE  PERSONNEL 

NAEVA  Project  Geophysicists:  Mr.  Alexander  Z.  Kostera 

Mr.  Leif  Riddervold 

XTECH  Data  Acquisition  Specialists:  Mr.  Vik  Banerjee 

Mr.  Mark  J.  Howard 

2.8  ATC’S  SURVEY  COMMENTS 

a.  Several  design  shortcomings  of  this  system  affect  both  safety  and  performance.  The 
bottom  of  the  sled  is  a  rectangular  platform  constructed  with  ABS  pipe.  Three  pipes  rode  on  the 
pond  bottom  parallel  to  the  direction  of  boat  travel;  two  were  dragged  perpendicular  to  the 
direction  of  travel  (beneath  the  EM  coils).  At  the  front  of  the  sled,  the  parallel  pipes  were  angled 
upward  with  a  perpendicular  support  (fig.  2  and  4).  The  configuration  permitted  objects  to  enter 
and  then  become  trapped  in  the  front  of  the  sled. 


Figure  4.  Side  view  of  sled. 
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The  vertical  component  of  this  sled  was  approximately  10  feet  high  and  was  also 
constructed  of  ABS  pipe  (fig.  2).  An  additional  4-foot  pipe  was  added  when  the  system 
surveyed  the  deeper  water  area  of  the  site  (white  pipe  visible  in  fig.  3).  Cement-filled 
pipes,  placed  inside  the  sled  runners,  served  two  purposes:  to  ensure  that  the  sled 
remained  on  the  bottom  and  to  lower  the  center  of  gravity  (fig.  5). 


Figure  5.  Cement  ballast. 


This  design  did  not  work  at  this  test  site.  The  platform  was  unstable,  particularly 
when  turning.  A  second  person  in  a  kayak  was  occasionally  needed  to  reorient  the  sled  in 
an  upright  position  (fig.  6). 


Figure  6.  Reorienting  the  survey  sled. 
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The  amount  of  ballast  placed  in  the  ABS  pipes  along  with  the  sled  design  also 
dislodged  an  undetermined  number  of  test  items  that  had  been  either  emplaced  on  the 
pond  bottom  or  pressed  into  the  sediment  to  be  flush  with  the  bottom.  The  first 
confirmation  that  this  was  happening  occurred  during  the  post-survey  processing  done  by 
NAEVA  at  the  test  site  (fig.  7).  An  ATC  geodetic/dive  team  attempted  to  locate  15 
randomly  selected  ground  truth  targets  after  XTECH’s  survey  (fig.  8).  All  15  were 
moved  from  their  locations.  Divers  reported  that  they  could  see  where  some  items  had 
been,  along  with  the  marks  left  by  the  sled. 


Figure  7.  Dragged  projectile. 
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Figure  8.  Checking  target  locations. 

At  an  actual  MEC  remediation  site,  this  type  of  system  would  increase  the  chances 
of  an  explosive  event.  A  2-meter  water  depth  and  the  length  of  the  towrope  would 
provide  a  limited  level  of  personal  protection,  depending  on  the  explosive  item,  but 
equipment  replacement  could  be  costly.  During  this  evaluation,  moving  the  ground  truth 
items  could  have  distorted  the  EM  signatures,  increased  the  percentage  of  false  positives 
or  background  alarm  calls  in  the  scoring  process  and  possibly  degraded  the  perfonnance 
evaluation  of  this  system. 

A  rope  that  was  looped  around  the  bow  of  the  boat  and  attached  at  two  points  on  the 
sled  pulled  the  sled.  Poles  mounted  on  the  stem  of  the  boat  that  extended  below  the 
waterline  prevented  the  rope  from  being  caught  by  the  outboard  motor  propeller.  This 
arrangement  worked  well  when  the  sled  was  pulled  in  a  straight  line  and  when  the  boat 
could  make  wide  turns.  However,  when  there  was  no  tension  on  the  towrope  and  the  boat 
was  maneuvering,  the  rope  could  get  into  the  propeller.  This  towing  system  did  not  work 
well  at  this  test  site. 


12 


SECTION  3.  SURVEY  COST  ANALYSIS 


3.1  DATES  OF  SURVEY 

The  NAEVA/XTECH  EM  system  was  tested  from  3  through  14  April  2006. 

3.2  SITE  CONDITIONS 

3.2.1  Atmospheric  Conditions 

An  ATC  weather  station  located  adjacent  to  the  test  site  recorded  the  average  temperature 
and  precipitation  on  an  hourly  basis  for  each  day  of  operation.  The  temperatures  listed  in 
Table  3-1  represent  the  average  temperature  from  0700  through  1700.  The  hourly  weather  logs 
used  to  generate  this  summary  are  provided  in  Appendix  A. 

3.2.2  Water  Conditions 


Water  conditions  were  monitored  using  a  TIDALITE  IV  Portable  Tide  Gauge  System  . 
Data  recorded  included  water  depth  and  temperature,  significant  wave  height  based  on  the 
average  1/3  wave  height  seen  over  the  test  period  using  the  Draper/Tucker  analysis  method,  and 
the  full-wave  frequency  calculated  by  full-wave  mean  crossing  detection.  The  values  displayed 
in  Table  3-1  were  averaged  from  0700  through  1700. 


TABLE  3-1 .  SITE  CONDITION  SUMMARY 


Date, 

06 

Air 

Temperature, 

°C 

Wind, 

km/h 

Water 

Temperature, 

°C 

Water 
Depth,  ma 

Significant 
Wave 
Height,  m 

Wave 

Frequency, 

Hz 

3  Apr 

13.1 

7.0 

10 

-0.1 

Lost 

Lost 

4  Apr 

10.2 

25.0 

10 

-0.1 

Lost 

Lost 

5  Apr 

6.5 

22.1 

10 

-0.1 

Lost 

Lost 

6  Apr 

10.7 

11.1 

10 

-0.1 

Lost 

Lost 

7  Apr 

12.5 

12.7 

10 

-0.1 

Lost 

Lost 

10  Apr 

11.4 

8.4 

10 

-0.2 

Lost 

Lost 

1 1  Apr 

13.9 

12.0 

10 

-0.2 

Lost 

Lost 

12  Apr 

16.0 

15.8 

10 

-0.2 

Lost 

Lost 

13  Apr 

19.7 

10.7 

10 

-0.2 

Lost 

Lost 

14  Apr 

16.7 

7.2 

11 

-0.2 

Lost 

Lost 

aVariance  between  the  required  2.4-meter  test  depth  and  actual  test  conditions. 
Lost  =  instrumentation  malfunction. 
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3.3  SURVEY  ACTIVITIES 


The  information  contained  in  this  section  provides  an  estimate  of  the  time  needed  and  costs 
associated  with  surveying  an  area  with  this  demonstrator’s  system.  This  includes  data  on 
equipment  setup  and  calibration,  site  survey  and  any  resurvey  time,  and  downtime  due  to  system 
malfunctions  and  maintenance  requirements. 

3.3.1  Survey  Times 

a.  A  government  representative  monitored  and  recorded  all  on-site  activities,  which  were 
grouped  into  one  of  1 1  categories.  The  first  eight  categories  were  chargeable  to  the  system  while 
the  last  three  were  not.  Categorizing  these  activities  provided  insight  into  the  technical  and 
logistical  aspects  of  the  system.  The  times  recorded  in  each  category  were  then  matched  with  the 
number  of  demonstrator  personnel,  assigned  skill  levels,  and  a  consistent  (across-vendor)  salary 
to  produce  an  estimate  of  the  survey  costs. 

(1)  Initial  setup/mobilization.  Started  at  the  time  the  demonstrator’s  equipment  arrived  at 
the  survey  site  and  stopped  when  the  system  was  ready  to  acquire  data. 

(2)  Daily  setup/close-up.  Monitored  time  spent  mounting  and  dismounting  the  equipment 
each  day. 

(3)  Instrument  calibration.  Recorded  the  amount  of  time  used  for  daily  quality  assurance 
checks  (e.g.,  sensors,  GPS  data,  survey  data  quality). 

(4)  Data  collection.  Time  spent  surveying  the  test  area. 

(5)  Downtime  (nonsurvey  time)  for  equipment/data  checks.  Covered  time  spent 
troubleshooting  equipment  or  verifying  survey  tracks. 

(6)  Downtime  (nonsurvey  time)  for  equipment  failure.  Examples  include  replacing 
damaged  cables,  lost  communication  with  base  station,  and  any  other  failure  that  prevented 
surveying.  Some  weather-related  failures  fall  into  this  category,  for  example,  light-emitting 
diode  (LED)  displays  darkened  by  the  sun,  wind  creating  waves  too  high  to  permit  surveying, 
etc. 


(7)  Downtime  (nonsurvey  time)  for  maintenance.  Battery  replacement  and  memory 
downloads  are  typical  examples. 

(8)  Demobilization.  Commenced  once  the  demonstrator  completed  the  survey  and 
concluded  the  final  on-site  check  of  the  test  data  and  ended  when  the  equipment  and  personnel 
were  ready  to  leave  the  site. 

(9)  Nonchargeable  downtime  for  breaks  and  lunch.  The  demonstrator’s  company  policy 
set  this  standard. 

(10)  Nonchargeable  downtime  for  weather-related  causes  (i.e.,  lightning,  high  wet-bulb 
heat  index,  and  similar  events). 
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(11)  Nonchargeable  downtime  due  to  ATC  range  operating  requirements.  Danger  zone 
conflicts,  lack  of  support  personnel,  equipment,  or  other  ATC-caused  delays. 

b.  Appendix  B  contains  the  daily  log  sheets.  Table  3-2  summarizes  that  information  to 
provide  insight  into  the  operational,  maintenance,  and  logistic  aspects  of  the  system. 


TABLE  3-2.  TIME  ON-SITE 


Date,  06 

3 

Apr 

4 

Apr 

5 

Apr 

6 

Apr 

7 

Apr 

10 

Apr 

11 

Apr 

12 

Apr 

13 

Apr 

14 

Apr 

Activity 

Totals, 

hr 

Activity  (daily  times  recorded  in  minutes) 

Initial  setup 

325 

205 

- 

- 

- 

- 

- 

- 

- 

- 

8.8 

Daily  setup/ 
close-up 

- 

30 

65 

60 

80 

70 

70 

100 

20 

30 

8.8 

Instrument 

calibration 

- 

55 

10 

10 

- 

5 

60 

5 

5 

15 

2.8 

Data 

collection 

- 

40 

- 

220 

- 

260 

320 

180 

275 

155 

24.2 

Equipment/ 
data  checks 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

0.0 

Equipment 

failure 

- 

40 

- 

85 

- 

135 

30 

240 

210 

- 

12.3 

Maintenance 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

0.0 

Demobilize 

- 

- 

- 

- 

- 

- 

- 

- 

- 

175 

2.9 

Breaks/ 

lunch 

30 

- 

- 

60 

- 

30 

60 

15 

- 

- 

3.3 

Weather- 

related 

80 

80 

345 

- 

340 

- 

- 

- 

- 

- 

14.1 

ATC 

downtime 

15 

- 

- 

- 

- 

- 

- 

- 

- 

- 

0.3 

Daily  total, 
hr 

7.5 

7.5 

6.8 

7.3 

7.0 

7.2 

9.0 

9.0 

8.5 

6.3 

Note:  Task  times  are  rounded  to  5 -minute  increments. 


3.3.2  On-Site  Data  Collection  Costs 


The  times  associated  with  the  1 1  activities  have  been  grouped  into  the  three  basic 
components  of  the  evaluation:  initial  setup,  site  survey,  and  pack-up  (demobilization).  Note  that 
site  survey  time  includes  daily  setup/stop  time,  data  collection,  breaks/lunch,  downtime  for 
equipment/data  checks  or  maintenance,  downtime  due  to  failure,  and  downtime  due  to  weather. 
This  combines  the  actual  survey  cost  with  the  demonstrator’s  associated  on-site  overhead  costs. 


15 


A  standardized  estimate  for  labor  costs  associated  with  this  effort  was  then  calculated 
using  the  following  job  categories:  supervisor  ($95. 00/hr),  data  analyst  ($57. 00/hr),  and  site 
support  ($28. 50/hr).  The  estimated  costs  are  presented  in  Table  3-3. 


TABLE  3-3.  CALCULATED  SURVEY  COSTS 


No.  of  Persons 

Hourly  Wage 

Hours 

Cost 

Initial  Setup 

Supervisor 

1 

$95.00 

8.8 

$836.00 

Data  analyst 

1 

$57.00 

8.8 

$501.60 

Site  support 

2 

$28.50 

8.8 

$501.60 

Subtotal 

$1,839.20 

Site  Survey 

Supervisor 

1 

$95.00 

48.1 

$4,569.50 

Data  analyst 

1 

$57.00 

48.1 

$2,741.70 

Site  support 

2 

$28.50 

48.1 

$2,741.70 

Subtotal 

$10,052.90 

Demobilization 

Supervisor 

1 

$95.00 

2.9 

$275.50 

Data  analyst 

1 

$57.00 

2.9 

$165.30 

Site  support 

2 

$28.50 

2.9 

$165.30 

Subtotal 

$606.10 

Total  on-site  costs 

$12,498.20 

3.4  COST  ANALYSIS 

The  data  collection  process  described  above  provided  an  on-site  cost  guide  to  compare  the 
performance  of  this  vendor  with  any  other  that  has  demonstrated  at  the  shallow  water  site.  It  is 
not  a  true  indicator  of  survey  costs.  Many  other  expenses  have  not  been  included,  such  as  travel 
costs,  per  diem,  off-site  data  processing  and  analysis,  company  overhead,  and  profit. 

Calculating  the  area  surveyed  was  done  by  plotting  the  raw  GPS  coordinates  and  then 
combining  the  sensor  swath  (line  spacing  and  associated  overlap). 

To  determine  the  number  of  acres  surveyed  per  day,  the  total  number  of  hours  spent  at  the 
test  site  (table  3-2)  was  divided  by  8  (converts  to  8-hr  days).  The  number  of  acres  was  then 
divided  by  the  number  of  8-hour  days.  The  cost  per  acre  was  determined  by  dividing  the  total 
survey  costs  (table  3-3)  by  the  same  number  of  acres.  This  infonnation  is  summarized  in 
Table  3-4. 
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TABLE  3-4.  SURVEY  COSTS 


Area  surveyed  (acre3) 

3.7 

Time  on-site  (8-hr  days) 

7.5 

Calculated  survey  cost  (U.S.  dollars) 

$12,489 

Acres  per  day 

0.49 

Cost  per  acre 

$3,378 

aAcre  =  4047  nr. 


Table  3-5  presents  a  comparison  of  Tetra  Tech’s  survey  costs  with  the  EQT-ORD  criteria. 


TABLE  3-5.  TEST  RESULTS  -  CRITERIA  COMPARISON 


Metric 

Threshold 

Objective 

NAEVA/XTECH 

Cost  rate 

$4000  per  acre 

$2000  per  acre 

$3378 

Production  rate 

5  acres  per  day 

50  acres  per  day 

0.49 
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SECTION  4.  TECHNICAL  PERFORMANCE  RESULTS 


4.1  AREA  SURVEYED 

4.1.1  Calculated  Area 


a.  Both  the  test  and  scoring  methodologies  required  the  demonstrator  to  survey 
100  percent  of  each  of  the  four  test  areas  (blind  grid,  open  water,  littoral,  and  deeper  water). 
Scoring  a  partially  surveyed  area  alters  the  ordnance  and  clutter  sample  sizes,  and  test  area 
boundaries,  and  decreases  the  statistical  confidence  in  the  perfonnance  statements  made  for  that 
area.  Allowing  partial  scoring  decreases  the  validity  of  performance  comparisons  made  between 
multiple  test  areas  for  a  single  demonstrator  and  comparisons  made  between  multiple 
demonstrators  for  a  single  test  area. 

b.  Realizing  that  some  systems  may  not  be  able  to  survey  100  percent  of  a  given  test  area, 
a  ranking  system  was  established.  The  percent  coverage  for  a  given  test  area  is  determined  by 
first  plotting  the  raw  GPS  coordinates  combined  with  the  sensor  swath  (line  spacing  and 
associated  overlap),  calculating  the  area  surveyed,  and  then  comparing  the  surveyed  area  with 
the  total  test  area. 


Section  Surveyed  x  100  =  %  Surveyed 
Test  Area  Size 

c.  The  demonstrator’s  system  is  always  scored  against  the  complete  ground  truth  for  a 
given  test  area  regardless  of  the  percentage  covered. 

4.1.2  Area  Assessment 

The  ranking  system  and  survey  results  are  presented  in  Table  4-1. 


TABLE  4- 1 .  M882  SURVEY  RANKING  SYSTEM  AND  RESULTS 


Ranking 

;  System 

Survey  Results 

Data  Use 

%  Area 
Covered 

Ranking 

Test  Area 

%  Area 
Covered 

95  to  100 

Met 

Blind  grid 

98 

Direct  comparison  between  systems  and 
areas. 

90  to  94 

Generally 

met 

Comparison  between  systems  and  areas.  A 
small  negative  bias  is  contained  in  the 
reported  numbers  (bias  not  quantified  in 
this  report). 

50  to  89 

Partially 

met 

Open  water 

84 

Reported,  not  compared  between  systems 
or  areas.  A  large  negative  bias  is 
contained  in  the  reported  numbers  (bias 
not  quantified  in  this  report). 

Deeper  water 

65 

0  to  49 

Not  met 

Littoral 

10 

Not  scored/not  reported. 
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4.2  SYSTEM  SCORING  PROCEDURES 


a.  The  scoring  entities  used  in  this  program  were  predicated  on  knowing  the  composition 
and  location  of  every  detectable  item  in  an  area.  The  deeper  water  area  is  the  one  exception. 
Ground  truth  targets  were  placed  in  this  area  without  a  pre-survey  and  clearing  operation. 
Therefore,  only  the  system’s  probability  of  detection  (Pd)  was  evaluated  in  this  area. 

b.  The  best  indicator  of  survey  perfonnance  is  the  blind  grid.  This  area  provides  a 
statically  valid,  controlled  environment  in  which  the  demonstrator  must  provide  a  response 
(ordnance,  clutter,  or  blank)  at  each  of  the  644  locations.  Comparison  of  the  response  and 
discrimination  lists  to  the  ground  truth  in  this  area  both  determines  the  range  of  ordnance  the 
system  can  reliably  detect  and  establishes  the  baseline  to  which  system  performance  in  all  other 
test  areas  is  measured. 

c.  The  scoring  terms  and  definitions,  along  with  an  explanation  of  the  receiver  operating 
characteristic  (ROC)  curve  development  and  the  chi-square  analysis  used  in  this  report,  are 
provided  in  Appendix  C. 

d.  Demonstrator  perfonnance  was  scored  in  two  stages:  response  and  discrimination. 

e.  Response  stage  scoring  evaluates  the  ability  of  the  demonstrator’s  system  to  detect 
emplaced  ground  truth  targets  without  regard  to  discriminating  ordnance  from  clutter.  In  this 
stage,  the  GPS  locations  and  signal  strengths  of  all  anomalies  the  demonstrator  deemed  sufficient 
for  further  investigation  and/or  processing  are  reported.  This  list  was  generated  with  minimal 
processing,  i.e.,  associating  signal  strength  with  GPS  location,  and  includes  only  signals  that  are 
above  the  system  noise  level. 

f.  The  discrimination  stage  evaluated  the  demonstrator’s  ability  to  segregate  ordnance 
from  clutter.  The  same  GPS  locations  reported  in  the  response  stage  anomaly  list  were  evaluated 
on  the  basis  of  the  demonstrator’s  discrimination  process  (section  2.6).  A  discrimination  stage 
list  was  generated  and  prioritized  on  the  basis  of  the  demonstrator’s  determination  that  an 
anomaly  was  more  likely  to  be  ordnance  rather  than  clutter.  Typically,  higher  output  values 
indicate  a  higher  confidence  that  an  ordnance  item  is  present  at  a  specified  location.  The 
demonstrator  then  specifies  the  threshold  value  for  the  prioritized  ranking  that  provides  optimal 
system  perfonnance.  This  value  is  the  discrimination  stage  threshold. 

g.  Both  the  response  and  discrimination  lists  contain  the  identical  number  of  potential 
target  locations,  differing  only  in  the  priority  ranking  of  the  declarations. 

h.  Within  both  of  these  stages,  the  following  entities  were  measured: 

(1)  Pd- 

(2)  Probability  of  false  positive  (Pfp). 

(3)  Probability  of  background  alarm  (Pba)/background  alann  rate  (BAR). 
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4.2.1  ROC  Curves 


a.  Based  on  the  entire  range  of  ground  truth  targets  used  at  this  site,  ROC  curves  were 
generated  for  both  the  response  and  discrimination  stages.  In  both  stages,  the  probability  of 
detection  versus  false  alarm  rates  was  plotted.  False  alarms  were  divided  into  two  groups: 
(1)  anomalies  corresponding  to  emplaced  clutter  items,  thereby  measuring  the  Pfp,  and  (2) 
anomalies  not  corresponding  to  any  known  item,  termed  background  alarms  (Pba)  in  the  blind 
grid  area  and  BAR  in  all  other  areas. 

b.  The  ROC  curves  for  the  response  and  discrimination  stages  for  all  areas  surveyed  are 
shown  in  Figures  9  through  12.  Horizontal  lines  illustrate  the  system  performance  at  the 
demonstrator’s  recommended  noise  level  during  the  response  stage,  or  discrimination  threshold 
level  in  the  discrimination  stage.  The  point  where  the  curve  crosses  the  horizontal  line  defines 
the  subset  of  targets  the  demonstrator  recommends  digging. 


Blind  Grid 


- Resp 

Disc 

- Noise 

—  Threshold 


Figure  9.  Blind  grid  Pd  versus  Pfp. 
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Blind  Grid 


Probability  of  Background  Alarm 


- Resp 

- Disc 

- Noise 

—  Threshold 


Figure  10.  Blind  grid  Pd  versus  Pba. 


Open  Water 


Figure  1 1 .  Open  water  Pd  versus  Pfp. 
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Probability  of  Detection  vs  Background  Alarm  Rate 


- Resp 

- Disc 

- Noise 

—  Threshold 


Background  Alarm  Rate 


Figure  12.  Open  water  Pd  versus  BAR. 


4.2.2  Detection  Results 


Detection  results,  broken  out  by  stage,  area  surveyed,  and  ordnance  size,  are  presented  in 
Table  4-2.  The  results  by  size  indicate  how  well  the  demonstrator  detected/discriminated 
ordnance  of  a  given  caliber.  Overall  results  summarize  ordnance  detection  over  a  given  area. 
All  values  were  calculated  assuming  the  number  of  detections  was  a  binomially  distributed 
random  variable.  These  results  are  reported  at  the  90  percent  reliability/95  percent  confidence 
levels  unless  otherwise  noted. 
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TABLE  4-2.  SYSTEM  DETECTION  SUMMARY 


By  Projectile  Caliber  | 

Metric 

Overall 

40  mm 

60  mm 

81  mm 

105  mm 

155  mm 

8  in.  j 

Blind  grid 

Response  stage 

Pd 

95.2% 

96.6% 

93.1% 

93.1% 

100.0% 

93.1% 

Pd  lower  90%  confidence 

92.0% 

87.2% 

82.7% 

82.7% 

92.4% 

82.7% 

Pfp 

92.0% 

Pfc  lower  90%  confidence 

88.6% 

Pba 

5.8% 

Discrimination  stage 

Pd 

45.5% 

27.6% 

44.8% 

37.9% 

55.2% 

62.1% 

Pd  lower  90%  confidence 

39.9% 

16.8% 

31.9% 

25.7% 

41.7% 

48.5% 

Pfp 

56.9% 

Pf„  lower  90%  confidence 

51.8% 

Pba 

1.8% 

Open  water 

Response  stage  \ 

Pd 

54.8% 

62.1% 

51.7% 

41.4% 

62.1% 

57.1% 

50.0% 

Pd  lower  90%  confidence 

49.3% 

48.5% 

38.4% 

28.8% 

48.5% 

44.9% 

20.1% 

Pfp 

54.2% 

Pf„  lower  90%  confidence 

49.4% 

BAR  m- 

0.016 

Discrimination  stage 

Pd 

31.8% 

24.1% 

24.1% 

20.7% 

55.2% 

37.1% 

16.7% 

Pd  lower  90%  confidence 

27.0% 

14.0% 

14.0% 

11.2% 

41.7% 

26.1% 

1.7% 

Pfp 

30.0% 

Pfc  lower  90%  confidence 

25.8% 

BAR  in'2 

0.007 

Littoral  region 

Response  stage 

Pd 

Test  area  not  surveyed 

Pd  lower  90%  confidence 

Pfp 

Pfp  lower  90%  confidence 

BAR  in- 

Discrimination  stage 

Pd 

Test  area  not  surveyed 

Pd  lower  90%  confidence 

Pfp 

Pfp  lower  90%  confidence 

BAR  m- 

Deeper  water 

|  Response  stage 

Pd 

24.1% 

24.1% 

Pd  lower  90%  confidence 

14.0% 

14.0% 

|  Discrimination  stage 

Pd 

24.1% 

24.1% 

Pd  lower  90%  confidence 

14.0% 

14.0% 

Response  stage  noise  level:  0.55 

Recommended  discrimination  threshold:  1 .5 
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4.2.3  System  Discrimination 


Using  the  demonstrator’s  recommended  setting,  the  items  detected  and  correctly  classified 
as  ordnance  were  further  evaluated  as  to  whether  the  demonstrator  could  correctly  identify  the 
ordnance  type.  The  list  of  ground  truth  ordnance  items  was  provided  to  the  demonstrator  before 
testing. 

NAEVA/XTECH’s  “dig  list”  discriminated  between  ordnance  and  clutter  but  not  between 
ordnance  types.  The  latter  was  an  optional  requirement. 

4.2.4  System  Effectiveness 

Efficiency  and  rejection  rates  were  calculated  to  quantify  the  discrimination  ability  at  two 
specific  points  of  interest  on  the  ROC  curve:  the  point  where  no  decrease  in  Pd  occurred 
(i.e.,  the  efficiency  is  by  definition  equal  to  1)  and  the  operator-selected  threshold.  These  values 
are  presented  in  Table  4-3. 


TABLE  4-3.  EFFICIENCY  AND  REJECTION  RATES 


Efficiency 

False  Positive 
Rejection  Rate 

Background  Alarm 
Rejection  Rate 

Blind  Grid 

At  operating  point 

0.48 

0.38 

0.68 

With  no  loss  of  Pa 

1.00 

0.38 

0.68 

Open  Water 

At  operating  point 

0.58 

0.45 

0.58 

With  no  loss  of  Pa 

1.00 

0.45 

0.58 

Littoral  Region 

At  operating  point 

Test  area  not  surveyed 

With  no  loss  of  Pa 

4.2.5  Chi-Square  Analysis 

A  chi-square  2x2  Contingency  Test  for  comparison  between  ratios  was  used  to  compare 
performance  across  test  areas  with  regard  to  Pdres,  Pddlsc,  Pfpres,  and  Pfpdlsc,  efficiency,  and  false 
alarm  rejection  rates.  The  intent  of  the  comparison  was  to  determine  whether  the  features 
introduced  in  each  test  region  had  a  degrading  effect  on  the  perfonnance  of  the  sensor  system. 

This  system  did  not  survey  enough  of  the  other  test  areas  to  permit  a  valid  comparison  of 
performance  between  the  areas. 
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4.2.6  Location  Accuracy 


The  data  points  in  the  scatter  graph  shown  in  Figure  13  represent  the  coordinates  of 
ordnance  items  in  the  open  water  test  area  that  were  first  detected  in  the  response  stage  within  a 
0.5 -me ter  radius  of  their  true  positions  and  then  correctly  identified  as  ordnance  in  the 
discrimination  stage.  The  maximum  error  represents  the  0.5-meter  detection  limit.  The  mean 
error  represents  the  statistical  mean  of  the  sample  considered. 


Positioning  Deltas 


Figure  13.  NAEVA/XTECH  open  water  positioning  deltas. 
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Comparisons  made  between  the  results  obtained  during  testing  and  the  EQT-ORD  criteria 
are  presented  in  Table  4-4. 


TABLE  4-4.  G882  TEST  RESULTS  -  CRITERIA  COMPARISON 


Metric 

Threshold 

Objective 

By  Area 

80%  ordnance  items 

95%  ordnance  items 

Blind  grid 

95.2% 

Detection 

buried  to  1  toot  and 
under  8  feet  (2.4  m)  of 

buried  to  4  feet  and  under 

8  feet  (2.4  m)  of  water. 

Open  water 

54.8% 

water. 

Littoral 

Not  surveyed 

Rejection  rate  of  50% 

Rejection  rate  of  90%  of 

Blind  grid 

38% 

of  emplaced  non-UXO 

emplaced  non-UXO 

Open  water 

45% 

Discrimination 

clutter. 

clutter. 

Littoral 

Not  surveyed 

Maximum  false 
negative  rate  of  10%. 

Maximum  false  negative 
rate  of  0.5%. 

Not  assessed.  An  analytical 
procedure  is  not  available  to 
address  this  criterion. 

Reacquisition 

Reacquire  within 

1  meter. 

Reacquire  within 

0.5  meter. 

The  number  of  correctly 
identified  items  is  insufficient  to 
draw  any  conclusions. 

Note:  The  blind  grid  and  open  water  areas  are  in  general  accordance  with  the  threshold 
requirements. 
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SECTION  5.  APPENDIXES 
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(Page  28  Blank) 


APPENDIX  A.  TEST  CONDITIONS  LOG 


ATMOSPHERIC  CONDITIONS 


Date,  06 


Time, 

EDT 

Average 

Wind 

Direction, 

deg 

Average 

Wind 

Speed, 

km/h 

Wind  Direction 
Average 
Standard 
Deviation,  deg 

Peak  Wind 
Speed, 
km/h 

Average 

Temperature, 

°C 

0700 

126 

6.6 

16 

12.6 

9.7 

0800 

154 

9.0 

16 

16.7 

10.2 

0900 

173 

5.2 

27 

10.6 

10.7 

1000 

111 

5.3 

19 

11.3 

10.7 

1100 

98 

7.9 

15 

15.9 

11.4 

1200 

96 

8.5 

16 

16.1 

12.4  j 

1300 

102 

8.1 

16 

19.5 

13.9  I 

1400 

134 

4.2 

34 

9.3 

15.1 

1500 

122 

5.5 

32 

13.2 

15.8 

1600 

137 

6.1 

27 

13.0 

17.2  j 

1700 

80 

10.1 

24 

20.1 

16.5 

0700 

292 

17.9 

11 

30.6 

7.8 

0800 

297 

22.7 

13 

39.1 

7.9 

0900 

297 

26.9 

12 

47.2 

7.8 

1000 

304 

27.0 

15 

46.4 

8.2 

1100 

313 

25.1 

16 

43.1 

9.0 

1200 

301 

23.3 

16 

40.9 

10.1 

1300 

297 

24.5 

18 

45.2 

11.3 

1400 

283 

24.6 

16 

50.1 

11.9 

1500 

286 

28.2 

14 

46.9 

12.5  I 

1600 

294 

28.2 

16 

51.0 

12.8 

1700 

287 

26.1 

17 

44.9 

13.0 

0700 

253 

8.1 

14 

16.3 

5.3 

0800 

227 

14.3 

29 

57.2 

5.8  ! 

0900 

293 

17.2 

12 

34.1 

4.1 

1000 

302 

26.7 

23 

58.6 

5.0 

1100 

318 

32.4 

14 

56.5 

4.2  | 

1200 

320 

27.7 

15 

53.9 

5.0  ! 

1300 

324 

24.8 

17 

44.6 

6.3 

1400 

318 

25.4 

16 

45.4 

7.8 

1500 

306 

24.0 

17 

43.0 

9.0 

1600 

315 

23.0 

16 

43.0 

9.1 

1700 

306 

19.8 

15 

35.9 

9.6 

0700 

261 

8.1 

8 

13.7 

6.3 

0800 

262 

8.5 

10 

15.0 

6.4 

0900 

275 

10.3 

12 

18.5 

7.4 

1000 

315 

10.1 

17 

17.4 

8.6 

1100 

336 

13.7 

19 

25.3 

9.3 

1200 

335 

13.4 

18 

25.3 

10.3 

1300 

320 

13.2 

18 

23.7 

11.6 

1400 

312 

12.6 

21 

26.6 

12.9 

A-l 


Date,  06 


Time, 

EDT 

Average 

Wind 

Direction, 

deg 

1500 

313 

1600 

308 

1700 

312 

0700 

105 

0800 

155 

0900 

191 

1000 

189 

1100 

197 

1200 

198 

1300 

196 

1400 

193 

1500 

199 

1600 

201 

1700 

203 

0700 

20 

0800 

43 

0900 

2 

1000 

21 

1100 

356 

1200 

333 

1300 

184 

1400 

172 

1500 

195 

1600 

190 

1700 

174 

0700 

57 

0800 

62 

0900 

126 

1000 

184 

1100 

186 

1200 

205 

1300 

200 

1400 

203 

1500 

200 

1600 

198 

1700 

201 

0700 

138 

0800 

144 

0900 

187 

1000 

196 

1100 

188 

1200 

193 

1300 

189 

1400 

180 

1500 

173 

1600 

180 

1700 

177 

Average 

Wind 

Speed, 

km/h 


12.7 


11.4 


7.9 


2.6 


1.8 


10.1 


15.0 


12.9 


16.3 


13.0 


16.7 


18.2 


17.4 


15.5 


6.1 


2.6 


6.1 


9.3 


9.0 


Wind  Direction 
Average 
Standard 
Deviation,  deg 


19 


21 


41 


14 


39 


11 


11 


13 


11 


12 


12 


12 


12 


12 


10 


22 


23 


26 


28 


25 


68 


37 


15 


15 


23 


2 


11 


41 


12 


17 


16 


15 


17 


12 


12 


11 


9 


15 


15 


14 


12 


11 


12 


12 


11 


12 


13 


Peak  Wind 
Speed, 
km/h 


26.2 


23.3 


17.2 


6.1 


5.6 


29.8 


24.2 


21.7 


26.2 


22.5 


31.1 


26.9 


28.2 


27.4 


10.5 


Average 

Temperature, 

°C 


14.2 


14.9 


15.6 


8.0 


A-2 


Date,  06 

Time, 

EDT 

Average 

Wind 

Direction, 

deg 

Average 

Wind 

Speed, 

km/h 

Wind  Direction 
Average 
Standard 
Deviation,  deg 

Peak  Wind 
Speed, 
km/h 

Average 

Temperature, 

°C 

13  Apr 

0700 

194 

6.6 

35 

17.1 

13.8 

0800 

208 

6.3 

18 

11.6 

13.7 

0900 

222 

7.9 

15 

13.4 

14.6 

1000 

233 

8.1 

25 

15.0 

16.2 

1100 

236 

9.2 

19 

14.2 

18.4 

1200 

263 

11.6 

52 

24.6 

20.7 

1300 

299 

13.8 

17 

23.7 

22.0 

1400 

283 

13.4 

19 

24.6 

23.4 

1500 

299 

12.6 

20 

26.7 

24.6  | 

1600 

314 

13.7 

19 

34.5 

24.8 

1700 

311 

15.0 

18 

29.0 

24.9 

14  Apr 

0700 

216 

6.6 

24 

11.3 

13.5 

0800 

166 

4.5 

53 

8.9 

13.9 

0900 

177 

5.0 

55 

14.3 

14.4 

1000 

16 

2.7 

79 

5.3 

16.0  ! 

1100 

162 

3.9 

72 

8.9 

16.6 

1200 

208 

7.1 

18 

10.3 

17.8 

1300 

242 

8.5 

26 

16.7 

16.8  j 

1400 

197 

10.8 

15 

19.6 

17.5  ! 

1500 

204 

11.4 

15 

21.6 

17.3 

1600 

184 

10.1 

15 

19.3 

18.7 

1700 

177 

8.4 

18 

14.3 

21.2 

Note:  The  TIDALITE  IV  Portable  Tide  Gauge  System®  was  not  operational.  Manual  water 
depth  and  temperature  measurements  were  recorded  each  morning.  The  single  measurements  for 
each  day  are  shown  in  Table  3-1. 
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B-l 


Company:  NAEVA/XTECH 

Date:  3  April  2006 

Personnel:  Leif  Riddervold,  Alexander 
Kostera,  Vik  Banerjee,  Scott  MacLellan 

Start 

Stop 

Remarks 

Activity 

Chargeable 

0830 

0845 

Arrived  at  site,  light  rain,  ATC  safety  briefing 

Downtime  ATC 

15 

0845 

0905 

Walked  around  pond. 

Initial  setup 

20 

0905 

1215 

Setup. 

Initial  setup 

190 

1215 

1245 

Lunch. 

Nonchargeable  downtime 

30 

1245 

1415 

Setup.  Rain  limited  the  amount  of  setup  that  could  be  done. 

Initial  setup 

90 

1415 

1440 

Rain  stopped  setup. 

Initial  setup 

25 

1440 

Left  site. 

1440 

1600 

Weather  delay. 

Weather  delay 

80 

Company:  NAEVA/XTECH 

Personnel:  Leif  Riddervold,  Alexander 

Date:  4  April  2006 

Kostera,  Vik  Banerjee,  Scott  MacLellan 

Start 

Stop 

Remarks 

Activity 

Chargeable 

0830 

1155 

Arrived  at  site.  Still  initial  setup.  High  winds  created  a  small-craft 
advisory. 

Initial  setup 

205 

1155 

1240 

Sled  in  water  calibrating  (winds  died  down  some). 

Calibration 

45 

1240 

1320 

Replaced  sections  of  rope  used  to  pull  the  sled. 

Downtime  equipment 

40 

1320 

1400 

Survey  attempt  at  collecting  one  or  two  lines.  Stopped  surveying  because 
of  high  winds. 

Data  collection 

40 

1400 

1410 

Calibration. 

Calibration 

10 

1410 

1440 

Cleanup. 

Daily  close-up 

30 

1440 

Left  site. 

1440 

1600 

Weather  delay. 

Weather  delay 

80 

APPENDIX  B.  DAILY  ACTIVITIES  LOG 
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Company:  NAEVA/XTECH 

Personnel:  Leif  Riddervold,  Alexander 

Date:  5  April  2006 

Kostera,  Vik  Banerjee,  Scott  MacLellan 

Start 

Stop 

Remarks 

Activity 

Chargeable 

0830 

0920 

Setup. 

Daily  setup 

50 

0920 

0930 

Static  calibration. 

Calibration 

10 

0930 

0945 

Canceled  for  the  day  because  of  current  squall  conditions  and  high  winds. 
Cleanup. 

Daily  close-up 

15 

0945 

1600 

Weather  delay. 

Weather  delay 

345 

Company:  NAEVA/XTECH 

Personnel:  Leif  Riddervold,  Alexander 

Date:  6  April  2006 

Kostera,  Vik  Banerjee,  Scott  MacLellan 

Start 

Stop 

Remarks 

Activity 

Chargeable 

0820 

0915 

Setup. 

Daily  setup 

55 

0915 

0925 

Survey. 

Data  collection 

10 

0925 

1000 

Poles  that  kept  the  towropes  clear  of  the  propeller  were  not  deep  enough  in 
the  water  to  accomplish  their  purpose.  Lengthening  the  submerged  poles 
solved  the  problem  in  the  deeper  water  but  created  a  “bottoming-out” 
problem  in  the  shallower  water. 

Downtime  equipment 

35 

1000 

1025 

Survey. 

Data  collection 

25 

1025 

1045 

Towropes  again  moved  under  the  boat  during  turns.  Problem  resolved  by 
attaching  floats  to  the  ropes. 

Downtime  equipment 

20 

1045 

1115 

Survey. 

Data  collection 

30 

1115 

1120 

Downloaded  survey  data. 

Calibration 

5 

1120 

1220 

Lunch. 

Nonchargeable  downtime 

60 

1220 

1330 

Survey. 

Data  collection 

70 

1330 

1340 

Problem  during  a  turn. 

Downtime  equipment 

10 

1340 

1505 

Survey. 

Data  collection 

85 

1505 

1510 

Static  calibration. 

Calibration 

5 

1510 

1530 

Sled  maintenance:  pulled  sled  from  the  water,  checked  zip  ties  and  the 
physical  condition  of  the  coils. 

Downtime  equipment 

20 

1530 

1630 

Cleanup. 

Daily  close-up 

60 
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Company:  NAEVA/XTECH 

Personnel:  Leif  Riddervold,  Alexander 

Date:  7  April  2006 

Kostera,  Vik  Banerjee,  Scott  MacLellan 

Start 

Stop 

Remarks 

Activity 

Chargeable 

0830 

0935 

Setup.  A  light  rain  was  falling;  winds  were  6  to  10  mph  with  gusts  to  16 
mph.  The  rain  was  sufficient  to  cause  concerns  about  the  electronics  in  the 
open  boat.  Limited  setup  was  done  while  the  weather  conditions  were 
monitored. 

Daily  setup 

65 

0935 

0935 

The  decision  was  made  to  cancel  because  of  the  weather  rather  then  risk 
damage  to  the  equipment. 

0935 

0950 

Secured  boat  for  the  weekend. 

Daily  close-up 

15 

0950 

1600 

Weather  delay. 

Weather  delay 

340 

Company:  NAEVA/XTECH 

Personnel:  Leif  Riddervold,  Alexander 

Date:  10  April  2006 

Kostera,  Vik  Banerjee,  Scott  MacLellan 

Start 

Stop 

Remarks 

Activity 

Chargeable 

0810 

0905 

Setup;  bailed  boat  from  weekend  rain. 

Daily  setup 

55 

0905 

0910 

Static  calibration. 

Calibration 

5 

0910 

1010 

Survey. 

Data  collection 

60 

1010 

1015 

Platform  tipped  onto  its  side  far  enough  to  submerge  the  GPS  antenna. 
System  appeared  okay. 

Downtime  equipment 

5 

1015 

1050 

Survey. 

Data  collection 

35 

1050 

1055 

Platform  tipped  again;  antenna  did  not  submerge. 

Downtime  equipment 

5 

1055 

1115 

Survey. 

Data  collection 

20 

1115 

1210 

Outboard  motor  problems:  one  cylinder’s  plug  was  fouling  at  the  low 
speeds  needed  to  pull  the  sled.  Mechanic  called. 

Downtime  equipment 

55 

1210 

1240 

Lunch. 

Nonchargeable  downtime 

30 

1240 

1255 

Changed  spark  plug. 

Downtime  equipment 

15 

1255 

1415 

Survey. 

Data  collection 

20 

1415 

1445 

Battery  on  charge  for  30  minutes. 

Downtime  equipment 

30 

1445 

1550 

Survey. 

Data  collection 

65 

1550 

1600 

Sled  appeared  to  have  too  much  drag;  returned  to  dock  and  pulled  sled  from 
the  water.  Everything  looked  all  right. 

Downtime  equipment 

10 

1600 

1700 

Survey. 

Data  collection 

60 

1700 

1715 

Noise  in  both  coil  channels.  Removed  sled  from  water  to  allow  the 
connections  to  dry  overnight. 

Downtime  equipment 

15 

1715 

1730 

Cleanup. 

Daily  close-up 

15 

1730 

Left  site. 
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Company:  NAEVA/XTECH 

Date:  11  April  2006 

Personnel:  Leif  Riddervold,  Alexander 
Kostera,  Vik  Banerjee,  Scott  MacLellan 

Start 

Stop 

Remarks 

Activity 

Chargeable 

0755 

0845 

Setup. 

Daily  setup 

50 

0845 

0855 

Static  calibration. 

Calibration 

10 

0855 

0905 

Survey. 

Data  collection 

10 

0905 

0915 

Problems  with  towropes. 

Downtime  equipment 

10 

0915 

0920 

Survey. 

Data  collection 

5 

0920 

0930 

Changed  plug  in  outboard  motor. 

Downtime  equipment 

10 

0930 

1010 

Survey. 

Data  collection 

40 

1010 

1020 

Nonsurvey,  conference  on  water. 

Downtime 

10 

1020 

1200 

Survey. 

Data  collection 

100 

1200 

1300 

Lunch;  recharged  battery. 

Nonchargeable  downtime 

60 

1300 

1320 

Loaded  survey  fdes. 

Calibration 

20 

1320 

1605 

Survey. 

Data  collection 

165 

1605 

1625 

Static  calibration. 

Calibration 

20 

1630 

1640 

Downloaded  survey  fdes. 

Calibration 

10 

1640 

1700 

Cleanup. 

Daily  close-up 

20 

Company:  NAEVA/XTECH 

Personnel:  Leif  Riddervold,  Alexander 

Date:  12  April  2006 

Kostera,  Vik  Banerjee,  Scott  MacLellan 

Start 

Stop 

Remarks 

Activity 

Chargeable 

0805 

0825 

Setup. 

Daily  setup 

35 

0825 

0900 

Attempted  positioning  of  the  sled  using  personnel  on  the  shore  to  survey 
the  littoral  area. 

Daily  setup 

35 

0900 

1300 

Troubleshooting  noise  in  coils. 

Downtime  equipment 

240 

1300 

1305 

Static  check. 

Calibration 

5 

1305 

1515 

Survey. 

Data  collection 

130 

1515 

1530 

Lunch. 

Non-chargeable 

downtime 

15 

1530 

1620 

Survey. 

Data  collection 

50 

1620 

1650 

Cleanup. 

Daily  close-up 

30 
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Company:  NAEVA/XTECH 

Date:  13  April  2006 

Personnel:  Leif  Riddervold,  Vik  Banerjee, 
Scott  MacLellan 

Start 

Stop 

Remarks 

Activity 

Chargeable 

0755 

1000 

Troubleshooting  coils;  decided  to  work  with  current  noise  levels. 

Downtime  equipment 

125 

1000 

1110 

Extended  the  length  of  the  12.5-foot  mast  by  3  feet  to  allow  operation  in 
deeper  water. 

Downtime  equipment 

70 

1110 

1115 

Static  calibration. 

Calibration 

5 

1115 

1445 

Survey. 

Data  collection 

210 

1445 

1500 

Refueled  boat. 

Downtime  equipment 

15 

1500 

1605 

Survey. 

Data  collection 

65 

1605 

1625 

Cleanup,  data  dump. 

Daily  close-up 

20 

Company:  NAEVA/XTECH 

Date:  14  April  2006 

Personnel:  Leif  Riddervold,  Vik  Banerjee, 
Scott  MacLellan 

Start 

Stop 

Remarks 

Activity 

Chargeable 

0810 

0840 

Setup. 

Daily  setup 

30 

0840 

0850 

Static  calibration. 

Calibration 

10 

0850 

1130 

Survey. 

Data  collection 

155 

1130 

1135 

Static  calibration. 

Calibration 

5 

1135 

1430 

Packed  up. 

Demobilization 

175 

APPENDIX  C.  TERMS  AND  DEFINITIONS 


GENERAL  DEFINITIONS 

Anomaly:  Location  of  a  system  response  deemed  to  warrant  further  investigation  by  the 
demonstrator  for  consideration  as  an  emplaced  ordnance  item. 

Detection:  An  anomaly  location  that  is  within  Rhai0  of  an  emplaced  ordnance  item. 

Munitions  and  Explosives  of  Concern  (MEC):  Specific  categories  of  military  munitions 
that  may  pose  unique  explosive  safety  risks,  including  UXO  as  defined  in  10  USC  101(e)(5), 
DMM  as  defined  in  10  USC  2710(e)(2)  and/or  munitions  constituents  (e.g.,  TNT,  RDX)  as 
defined  in  10  USC  2710(e)(3)  that  are  present  in  high  enough  concentrations  to  pose  an 
explosive  hazard. 

Emplaced  Ordnance:  An  ordnance  item  buried  by  the  government  at  a  specified  location 
in  the  test  site. 

Emplaced  Clutter:  A  clutter  item  (i.e.,  nonordnance  item)  buried  by  the  government  at  a 
specified  location  in  the  test  site. 

Rhai0:  A  predetermined  radius  about  the  periphery  of  an  emplaced  item  (clutter  or 
ordnance)  within  which  a  location  identified  by  the  demonstrator  as  being  of  interest  is 
considered  to  be  a  response  from  that  item.  For  the  purpose  of  this  program,  a  circular  halo  0.5 
meters  in  radius  will  be  placed  around  the  center  of  the  object  for  all  clutter  and  ordnance  items 
less  than  0.6  meters  in  length.  When  ordnance  items  are  longer  than  0.6  meters,  the  halo 
becomes  an  ellipse  where  the  minor  axis  remains  1  meter  and  the  major  axis  is  equal  to  the 
projected  length  of  the  ordnance  onto  the  ground  plane  plus  1  meter. 

Response  Stage  Noise  Level:  The  level  that  represents  the  point  below  which  anomalies 
are  not  considered  detectable.  Demonstrators  are  required  to  provide  the  recommended  noise 
level  for  the  blind  grid  test  area. 

Discrimination  Stage  Threshold:  Demonstrator’s  select  the  threshold  level  that  they 
believe  provides  optimum  performance  of  the  system  by  retaining  all  detectable  ordnance  and 
rejecting  the  maximum  amount  of  clutter.  This  level  defines  the  subset  of  anomalies  the 
demonstrator  would  recommend  digging  based  on  discrimination. 

Binomially  Distributed  Random  Variable:  A  random  variable  of  the  type  that  has  only  two 
possible  outcomes,  say  success  and  failure,  and  is  repeated  for  n  independent  trials  with  the 
probability  p  of  success  and  the  probability  1-p  of  failure  being  the  same  for  each  trial.  The 
number  of  successes  x  observed  in  the  n  trials  is  an  estimate  of  p  and  is  considered  to  be  a 
binomially  distributed  random  variable. 
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RESPONSE  STAGE  DEFINITIONS 


Response  Stage  Probability  of  Detection  (Pdres):  Pdres  =  (No.  of  response  stage  detections)/ 
(No.  of  emplaced  ordnance  in  the  test  site). 

Response  Stage  False  Positive  (fpres):  An  anomaly  location  that  is  within  Rhai0  of  an 
emplaced  clutter  item. 

Response  Stage  Probability  of  False  Positive  (Pfpres):  Pfpres  =  (No.  of  response  stage  false 
positives)/(No.  of  emplaced  clutter  items). 

Response  Stage  Background  Alarm:  An  anomaly  in  a  blind  grid  cell  that  contains  neither 
emplaced  ordnance  nor  an  emplaced  clutter  item.  An  anomaly  location  in  the  open  water  or 
littoral  scenarios  that  is  outside  Rhaio  of  any  emplaced  ordnance  or  emplaced  clutter  item. 

Response  Stage  Probability  of  Background  Alann  (Pbares):  blind  grid  only:  Pbares  =  (No.  of 
response  stage  background  alarms)/(No.  of  empty  grid  locations). 

Response  Stage  Background  Alarm  Rate  (BARres):  open  water  only:  BARres  =  (No.  of 
response  stage  background  alarms)/(arbitrary  constant). 

Note  that  the  quantities  Pd'es,  Prp'cs,  Pbares,  and  BARres  are  functions  of  tres,  the  threshold 
applied  to  the  response  stage  signal  strength.  These  quantities  can,  therefore,  be  written  as 

Pdres(tres),  PfPres(tres),  Pbares(tres),  and  BARres(tres). 

DISCRIMINATION  STAGE  DEFINITIONS 

Discrimination:  The  application  of  a  signal  processing  algorithm  or  human  judgment  to 
response  stage  data  that  discriminates  ordnance  from  clutter.  Discrimination  should  identify 
anomalies  that  the  demonstrator  has  high  confidence  correspond  to  ordnance,  as  well  as  those 
that  the  demonstrator  has  high  confidence  correspond  to  nonordnance  or  background  returns. 
The  former  should  be  ranked  with  highest  priority  and  the  latter  with  lowest. 

Discrimination  Stage  Probability  of  Detection  (Pddlsc):  Pddlsc  =  (No.  of  discrimination  stage 
detections)/(No.  of  emplaced  ordnance  in  the  test  site). 

Discrimination  Stage  False  Positive  (fpdlsc):  An  anomaly  location  that  is  within  Rhai0  of  an 
emplaced  clutter  item. 

Discrimination  Stage  Probability  of  False  Positive  (Pfpdlsc):  P|pdlsc  =  (No.  of  discrimination 
stage  false  positives)/(No.  of  emplaced  clutter  items). 

Discrimination  Stage  Background  Alarm:  An  anomaly  in  a  blind  grid  cell  that  contains 
neither  emplaced  ordnance  nor  an  emplaced  clutter  item.  An  anomaly  location  in  the  open  water 
or  littoral  scenarios  that  is  outside  Rhai0  of  any  emplaced  ordnance  or  emplaced  clutter  item. 
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Discrimination  Stage  Probability  of  Background  Alarm  (Pbadlsc):  Pbadlsc  =  (No.  of 
discrimination  stage  background  alarms)/(No.  of  empty  grid  locations). 

Discrimination  Stage  Background  Alann  Rate  (BARdlsc):  BARdlsc  =  (No.  of  discrimination 
stage  background  alarms)/(arbitrary  constant). 

Note  that  the  quantities  Pddlsc,  Pfpdisc,  Pbadlsc,  and  BARdlsc  are  functions  of  tdlsc,  the  threshold 
applied  to  the  discrimination  stage  signal  strength.  These  quantities  can,  therefore,  be  written  as 
Pddisc(td|sc),  Pfpdlsc(tdisc),  Pbadisc(tdisc),  and  BARdisc(tdisc). 

RECEIVER  OPERATING  CHARACERISTIC  (ROC)  CURVES 

ROC  curves  at  both  the  response  and  discrimination  stages  can  be  constructed  on  the  basis 
of  the  above  definitions.  The  ROC  curves  plot  the  relationship  between  Pd  versus  Pfp  and  Pd 
versus  BAR  or  Pba  as  the  threshold  applied  to  the  signal  strength  is  varied  from  its  minimum 
(tmin)  to  its  maximum  (tmax)  value.1  Figure  A-l  shows  how  Pd  versus  Pfp  and  Pd  versus  BAR  are 
combined  into  ROC  curves.  Note  that  the  “res”  and  “disc”  superscripts  have  been  suppressed 
from  all  the  variables  for  clarity. 


Figure  A-l.  ROC  curves  for  open-site  testing.  Each  curve  applies  to  both  the  response  and 
discrimination  stages. 


'Strictly  speaking,  ROC  curves  plot  the  Pd  versus  Pba  over  a  predetermined  and  fixed  number  of 
detection  opportunities  (some  of  the  opportunities  are  located  over  ordnance  and  others  are 
located  over  clutter  or  blank  spots).  In  an  open  water  scenario,  each  system  suppresses  its  signal 
strength  reports  until  some  bare-minimum  signal  response  is  received  by  the  system. 
Consequently,  the  open  water  ROC  curves  do  not  have  infonnation  from  low-signal  output 
locations,  and,  furthermore,  different  contractors  report  their  signals  over  a  different  set  of 
locations  on  the  ground.  These  ROC  curves  are  thus  not  true  to  the  strict  definition  of  ROC 
curves  as  defined  in  textbooks  on  detection  theory.  Note,  however,  that  the  ROC  curves 
obtained  in  the  blind  grid  test  sites  are  true  ROC  curves. 
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METRICS  TO  CHARACTERIZE  THE  DISCRIMINATION  STAGE 


The  demonstrator  is  also  scored  on  efficiency  and  rejection  ratio,  which  measure  the 
effectiveness  of  the  discrimination  stage  processing.  The  goal  of  discrimination  is  to  retain  the 
greatest  number  of  ordnance  detections  from  the  anomaly  list  while  rejecting  the  maximum 
number  of  anomalies  arising  from  non  ordnance  items.  The  efficiency  measures  the  amount  of 
detected  ordnance  retained  by  the  discrimination,  while  the  rejection  ratio  measures  the  fraction 
of  false  alarms  rejected.  Both  measures  are  defined  relative  to  the  entire  response  list,  i.e.,  the 
maximum  ordnance  detectable  by  the  sensor  and  its  accompanying  false  positive  rate  or 
background  alann  rate. 

Efficiency  (E):  E  =  Pddlsc(tdlsQ/Pdres(tminres):  measures  (at  a  threshold  of  interest),  the  degree 
to  which  the  maximum  theoretical  detection  performance  of  the  sensor  system  (as  determined  by 
the  response  stage  tm,n)  is  preserved  after  application  of  discrimination  techniques.  Efficiency  is 
a  number  between  0  and  1 .  An  efficiency  of  1  implies  that  all  of  the  ordnance  initially  detected 
in  the  response  stage  was  retained  at  the  specified  threshold  in  the  discrimination  stage,  tdls0. 

False  Positive  Rejection  Rate  (Rfp):  Rrp  =  1  -  [ P ipd ' "c ( td  1  sc )/ P rp'es( t m , n 'cs ) ] :  measures  (at  a 
threshold  of  interest),  the  degree  to  which  the  sensor  system’s  false  positive  performance  is 
improved  over  the  maximum  false  positive  performance  (as  determined  by  the  response  stage 
tmin).  The  rejection  rate  is  a  number  between  0  and  1.  A  rejection  rate  of  1  implies  that  all 
emplaced  clutter  initially  detected  in  the  response  stage  was  correctly  rejected  at  the  specified 
threshold  in  the  discrimination  stage. 

Background  Alann  Rejection  Rate  (Rba): 

Blind  Grid:  Rba  =  1  -  [Pbadlsc(tdlsc)/Pbares(tmmres)] 

Open  Water:  Rba  =  1  -  [BARdlsc(tdlsc)/BARres(tramres)]) 

Measures  the  degree  to  which  the  discrimination  stage  correctly  rejects  background  alarms 
initially  detected  in  the  response  stage.  The  rejection  rate  is  a  number  between  0  and  1.  A 
rejection  rate  of  1  implies  that  all  background  alarms  initially  detected  in  the  response  stage  were 
rejected  at  the  specified  threshold  in  the  discrimination  stage. 

CHI-SQUARE  COMPARISON  EXPLANATION 

The  chi-square  test  for  differences  in  probabilities  (or  2  x  2  contingency  table)  is  used  to 
analyze  two  samples  drawn  from  two  different  populations  to  see  if  both  populations  have  the 
same  or  different  proportions  of  elements  in  a  certain  category.  More  specifically,  two  random 
samples  are  drawn,  one  from  each  population,  to  test  the  null  hypothesis  that  the  probability  of 
event  A  (some  specified  event)  is  the  same  for  both  populations  (ref  3,  pages  144  through  151). 

A  one-sided  2x2  contingency  table  is  used  in  the  Shallow  Water  Site  Program  to  compare 
each  area  (open  water,  littoral,  deep  water)  to  the  blind  grid  since  each  area  introduces  a  water 
feature  that  makes  it  potentially  more  difficult  to  survey  than  the  blind  grid.  The 
one-sided  2x2  contingency  table  is  used  to  determine  if  there  is  reason  to  believe  that  the 
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proportion  of  ordnance  correctly  detected/discriminated  by  demonstrator  X’s  system  is 
significantly  degraded  by  the  more  challenging  feature  introduced.  A  two-sided  2x2 
contingency  table  is  used  to  compare  performance  between  any  two  of  the  test  sites  other  than 
the  blind  grid  to  determine  if  there  is  reason  to  believe  that  the  proportion  of  ordnance  correctly 
detected/discriminated  by  demonstrator  X’s  system  is  significantly  different  between  those  two 
test  sites. 

The  test  statistic  of  the  2x2  contingency  table  is  the  chi-square  distribution  with  one 
degree  of  freedom.  For  the  one-sided  test,  a  significance  level  of  0.05  is  chosen,  which  sets  a 
critical  decision  limit  of  3.84  from  the  chi-square  distribution  with  one  degree  of  freedom.  It  is  a 
critical  decision  limit  because  if  the  test  statistic  calculated  from  the  data  exceeds  this  value,  the 
two  proportions  tested  will  be  considered  significantly  different.  If  the  test  statistic  calculated 
from  the  data  is  less  than  this  value,  the  two  proportions  tested  will  be  considered  not 
significantly  different. 

An  exception  must  be  applied  when  either  a  0  or  100  percent  success  rate  occurs  in  the 
sample  data.  The  chi-square  test  cannot  be  used  in  these  instances.  Instead,  Fisher’s  Exact  Test 
is  used,  and  the  critical  decision  limit  is  the  chosen  significance  level,  which  is  0.05  for 
one-sided  tests  and  0.10  for  two-sided  tests.  With  Fisher’s  test,  if  the  test  statistic  (p-value)  is 
less  than  the  critical  value,  then  the  null  hypothesis  of  similar  performance  is  rejected  in  favor  of 
the  alternative  hypothesis:  significantly  greater  than  for  the  one-sided  case  or  significantly 
different  for  the  two-sided  case. 

Shallow  Water  UXO  Detection  Test  Site  examples,  where  blind  grid  results  are  compared 
to  those  from  the  open  water  and  littoral  sites  and  the  nongrid  sites  (open  water  and  littoral),  are 
compared  to  each  other  as  follows.  It  should  be  noted  that  a  significant  result  does  not  prove  a 
cause  and  effect  relationship  exists  between  the  change  in  survey  area  and  sensor  perfonnance; 
however,  it  does  serve  as  a  tool  to  indicate  that  one  data  set  reflects  relatively  degraded  system 
performance  of  a  large  enough  scale  than  can  be  accounted  for  merely  by  chance  or  random 
variation.  Note  also  that  a  result  that  is  not  significant  indicates  that  there  is  not  enough  evidence 
to  declare  that  anything  more  than  chance  or  random  variation  within  the  same  population  is  at 
work  between  the  two  data  sets  being  compared. 

Demonstrator  X  achieves  the  following  overall  results  after  surveying  each  of  the  three 
areas  using  the  same  system  (results  indicate  the  number  of  ordnance  detected  divided  by  the 
number  of  ordnance  emplaced): 

Blind  grid  Open  water  Littoral 

Pdres  100/100  =  1.0  8/10  =  .80  20/33=  .61 

Pddisc  80/100  =  0.80  6/10  =  .60  8/33  =  .24 

Pdres:  BLIND  GRID  versus  OPEN  WATER.  Using  the  example  data  above  to  compare 
probabilities  of  detection  in  the  response  stage,  all  100  ordnance  out  of  100  emplaced  ordnance 
items  were  detected  in  the  blind  grid  while  8  ordnance  out  of  10  emplaced  were  detected  in  the 
open  water.  Fisher’s  test  must  be  used  since  a  100  percent  success  rate  occurs  in  the  data. 
Fisher’s  test  uses  the  four  input  values  to  calculate  a  test  statistic  (p-value)  of  0.0075  that  is 
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compared  against  the  critical  value  of  0.05.  Since  the  test  statistic  is  less  than  the  critical  value, 
the  smaller  response  stage  detection  rate  (0.80)  is  considered  to  be  significantly  less  at  the 
0.05  level  of  significance.  While  a  significant  result  does  not  prove  a  cause  and  effect 
relationship  exists  between  the  change  in  survey  area  and  degradation  in  performance,  it  does 
indicate  that  the  detection  ability  of  demonstrator  X’s  system  seems  to  have  been  degraded  in  the 
open  water  relative  to  results  from  the  blind  grid  using  the  same  system. 

Pddlsc:  BLIND  GRID  versus  OPEN  WATER.  Using  the  example  data  above  to  compare 
probabilities  of  detection  in  the  discrimination  stage,  80  out  of  100  emplaced  ordnance  items 
were  correctly  discriminated  as  ordnance  in  blind  grid  testing  while  6  out  of  10  emplaced 
ordnance  items  were  correctly  discriminated  as  such  in  open  water  testing.  Those  four  values  are 
used  in  the  chi-square  Contingency  Test  to  calculate  a  test  statistic  of  1.12.  Since  the  test 
statistic  is  less  than  the  critical  value  of  3.84,  the  two  discrimination  stage  detection  rates  are 
considered  to  be  not  significantly  different  at  the  0.05  level  of  significance. 

Pdres:  BLIND  GRID  versus  LITTORAL.  Using  the  example  data  above  to  compare 
probabilities  of  detection  in  the  response  stage,  100  out  of  100  and  20  out  of  33  are  used  to 
calculate  a  test  statistic  (<  0.000)  that  is  compared  against  the  critical  value  of  0.05.  Since  the 
test  statistic  is  less  than  the  critical  value,  the  smaller  response  stage  detection  rate  (0.61)  is 
considered  to  be  significantly  less  at  the  0.05  level  of  significance. 

Pddlsc:  BLIND  GRID  versus  LITTORAL.  Using  the  example  data  above  to  compare 
probabilities  of  detection  in  the  discrimination  stage,  80  out  of  100  and  8  out  of  33  emplaced 
ordnance  items  were  correctly  discriminated  as  such  in  open  water  testing.  Those  four  values  are 
used  to  calculate  a  test  statistic  of  32.01.  Since  the  test  statistic  is  greater  than  the  critical  value 
of  3.84,  the  smaller  discrimination  stage  detection  rate  (0.24)  is  considered  to  be  significantly 
less  at  the  0.05  level  of  significance. 

Pdres:  OPEN  WATER  versus  LITTORAL.  Using  the  example  data  above  to  compare 
probabilities  of  detection  in  the  response  stage,  8  out  of  10  and  20  out  of  33  are  used  to  calculate 
a  test  statistic  of  0.56.  Since  the  test  statistic  is  less  than  the  critical  value  of  2.71,  the  two 
response  stage  detection  rates  are  considered  to  be  not  significantly  different  at  the  0.10  level  of 
significance. 

Pddlsc:  OPEN  WATER  versus  LITTORAL.  Using  the  example  data  above  to  compare 
probabilities  of  detection  in  the  discrimination  stage,  6  out  of  10  and  8  out  of  33  are  used  to 
calculate  a  test  statistic  of  2.98.  Since  the  test  statistic  is  greater  than  the  critical  value  of  2.71, 
the  two  discrimination  stage  detection  rates  are  considered  to  be  significantly  different  at  the 
0.10  level  of  significance.  While  a  significant  result  does  not  prove  a  cause  and  effect 
relationship  exists  between  the  change  in  survey  area  and  change  in  performance,  it  does  indicate 
that  the  ability  of  Demonstrator  X  to  correctly  discriminate  seems  to  have  been  degraded  by 
features  of  the  littoral  area  relative  to  results  from  the  open  water  using  the  same  system. 
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APPENDIX  E.  ABBREVIATIONS 


ABS 

ADST 

APG 

ATC 

BAA 

BAR 

DMM 

EM 

EQT 

EQT-ORD  = 

ERDC 

ESTOP 

GPS 

LED 

MEC 

METDC  = 

Pba 

Pd 

Pdisc 

d 

Pdres 

pfpdlsc 

Pfp 

Pfpres 

POC 

QA 

QC 

ROC 

SERDP  = 
USAEC  = 

uxo 


acrylonitrile-butadiene-styrene 
Aberdeen  Data  Services  Team 
Aberdeen  Proving  Ground 
U.S.  Army  Aberdeen  Test  Center 
Broad  Agency  Announcement 
background  alann  rate 
discarded  military  munitions 
electromagnetic 

Army  Environmental  Quality  Technology  Program 

Environmental  Quality  Technology  -  Operational  Requirements  Document 

U.S.  Army  Corps  of  Engineers  Engineering,  Research  and  Development  Center 

Environmental  Security  Technology  Certification  Program 

Global  Positioning  System 

light-emitting  diode 

munitions  and  explosives  of  concern 

Military  Environmental  Technology  Demonstration  Center 

probability  of  background  alann  rate 

probability  of  detection 

probability  of  detection,  discrimination  stage 

probability  of  detection,  response  stage 

probability  of  false  positive 

probability  of  false  positive,  discrimination  stage 

probability  of  false  positive,  response  stage 

point  of  contact 

quality  assurance 

quality  control 

receiver  operating  characteristic 

Strategic  Environmental  Research  and  Development  Program 
U.S.  Army  Environmental  Command 
unexploded  ordnance 
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